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Introduction 


In 2014, the Center for Research on Education Outcomes at Stanford University released 
a report Charter School Performance in California. The report presented a comprehensive 
examination of the impact of charter school enrollment on the academic progress of the 
students who attended them during the school years 2005-2006 through 2010-2011. 


This memo recaps the major findings of the 2014 report, addresses the factors that have 
inhibited a more contemporary view of charter school performance and provides a brief epilog 
to the former study as a pathway to future investigations. 


About the 2014 Report 


The 2014 report is included in full in this briefing memorandum as an Appendix. To 
build the context for the updated analysis that is the key focus of this memorandum, it is 
helpful to review the approach and major findings of the 2014 report. 


CREDO used student-level education records obtained from the California Department 
of Education under a data sharing Memorandum of Understanding to create statistical models 
of see how well students in California charter schools are learning compared to the experience 
they would have otherwise had in their local traditional public schools (TPS). The outcome of 
interest is the amount of progress a typical charter school student makes in a year’s time 
compared to their TPS peers. 


It takes two successive snapshots of academic achievement (status) to build a measure 
of academic progress from one year to the next (growth). Growth is a preferred measure 
because it allows school impacts to be separated from other things that might affect a student’s 
achievement at a single point in time, such as maturation or differences in student background. 
From six years of student data, we are able to create five growth periods, though not all 
students will be observed the entire time. 


CREDO uses a matching method called the Virtual Control Record that identified 
students in local TPS who exactly match charter school students, including prior academic 
baseline, to create for each charter student an estimate of what learning they would have 
gained had they attended their local TPS.* That alternative scenario is based on the observed 
experience of real students which is averaged to build a “virtual twin.” 


Student performance was examined separately for reading and math based on the 
students’ scores on California’s standardized STAR exams. The effect for charter school 
learning can then be compared to the learning in TPS. 


1 A helpful infographic of how CREDO identifies comparison students can be seen at 
http://credo.stanford.edu/virtual-control-records/. 
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Summary of Key Findings of 2014 Report 


Since the California charter school legislation was passed in 1991, interest in their performance 
has been keen. The 2014 report is the latest study of performance using student-level data, 
which provides a highly accurate estimate of their effectiveness. Across the five growth periods 
covered by this study, CREDO estimated the overall average impact on one-year academic 
growth for California’s charter school students. The “topline” results are presented in Figure 1, 
which is copied directly from the 2014 report. 


Figure 1: Average Learning Gains in California Charter Schools Compared to Gains for VCR 
Students in Each Charter Schools’ Feeder TPS 
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Before turning to the actual results, a brief discussion of the format used in Table 1 is 
advisable. The research estimated the one-year academic growth of students of charter school 
students and their matched comparison peers in traditional public schools (TPS) and then 
compared them. The results are presented as the marginal performance of charter students, 
on average, against the average academic gains of TPS students, represented as the “zero” line 
in the figure. The statistical model used in the calculation controlled for differences in student 
characteristics, so the results reflect just the effect of school of enrollment and are not 
influenced by differences in demographics from school to school. 
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The results are expressed in standard deviation units, which are meaningful to 
researchers but few others. Those values are presented in the left-hand y-axis in Table 1. 
Positive values mean that charter students realized more academic progress in a year’s time 
than their TPS counterparts; negative values mean they made less progress than their TPS peers 
per year. 


To better explain the findings, we transposed the standard deviation units into “days of 
learning” by considering a typical year of schooling for the average TPS student in this study to 
be 180 days. If the growth of charter school students was positive, it is as though they 
attended school for 180 day plus additional days. If negative, the reverse is the case: it is as 
though they attended fewer days than the 180-day standard. 


Figure 1 shows that in California across the five growth periods under study, students 
enrolled in charter schools realized more academic progress in reading than their matched TPS 
comparisons, amounting to about 14 additional days of learning. In math, the results were 
reversed: charter school students made 14 days less progress in a year’s time than their TPS 
matched cases. 


These averages represent the results that a typical student in a typical charter school 
would realize. Of course, each of these averages has a distribution of results around it. Some 
students will have gains that are significantly more positive than the average and some will 
have results that are significantly inferior. The distribution of results around the average raises 
questions about how similar the patterns are for different types of students. 


Table 1: Relative Growth of Student Groups Compared to their TPS Virtual Twins 


Reading 


Days of Math Days of 
Student Group Reading? Learning® Learning? 


Charter Black 


Charter Black Poverty 05** 36 .06** 43 
Charter Hispanic .01** 7 -.02** -14 
Charter Hispanic Poverty .03** 22 .04** 29 
Charter White -.01** -7 -.10** -72 
Charter Asian -.01** -7 -.04** -29 
Charter Poverty 2** 14 0a7* 29 
Charter Special Education 2? 14 Ole" 7 


Charter English Learner 


* Value represents effect measured in standard deviation units. 
> Days of learning transforms the growth effect to days gained or lost based on a 180-day school year. 
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Many California charter schools aim to address students who have historically not fared 
wellin TPS. Looking at learning impacts experienced by different student groups allows us to 
see if these efforts are succeeding. Table 1 above presents a summary of the marginal impact 
of charter school enrollment for each of the student sub-groups, relative to their comparison 
peers in traditional public schools. [This table was created specifically for this briefing 
memorandum and does not appear in the original report.] 


In California, students who are Black or Hispanic or are in poverty made significantly 
greater progress in charter schools than their TPS peers. The one exception was seen when the 
learning gains for all Hispanic charter school students were estimated. However, when the 
group of Hispanic charter school students were further distinguished by their poverty status, 
Hispanic students in poverty were found to be significantly advantaged by attending charter 
school students. Even though Black students in general were found to have improved 
outcomes in charter schools, the academic progress of Black students in poverty was found to 
be even more positively affected by attending a charter school. 


California charter schools provide greater learning gains for students who bring with 
them attributes that might be expected to adversely impact their academic progress. Students 
who are identified as needing Special Education supports fare slightly better in charter schools 
than their matched peers in the TPS the charter students would otherwise have attended. For 
students who are learning English while attending school, the charter school advantage is large 
and significant. It also bears noting that this finding is not typical in other states, so there may 
be an opportunity to learn what approach these schools take with their English learners for 
possible replication in other schools. 


Two student groups did not fare as well by attending charter schools: Asian students 
and White students. In both cases, their math performance was negatively impacted to a 
dramatic degree compared to the gains their matched peers made in TPS. These differences 
are explained in part by the differences in location of charter schools serving significant 
proportions of Whites and Asians; these schools are found in suburban and rural communities 
where the absolute achievement of the TPS schools in higher than is found in urban TPS 
schools. 


Student Growth and Achievement 


While student academic growth is the preferred way to examine the impact of schools 
on student performance, achievement is also important as a signal of what students actually 
know. Both factors provide important signals about the strength of schools, particularly when 
considered dynamically. By displaying achievement and growth together we can tell the story 
about schools in a more nuanced way. 


We built a graph to display achievement and growth of the California charter schools 
simultaneously. The Y-axis represents the 50" percentile of achievement in the state; schools 


that appear above the line are in the top half of achievement in the state, and those below are 
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achievement below the state average. The X-axis shows average annual growth for the school 
relative to its local alternatives. Schools in the right half of the charter display greater-than- 
average growth while those on the left do not produce as much growth as their peer schools. 


The separate quadrants of the table tell different stories about the schools that fall 
within them. The upper right quadrant is the highly desired combination of above average 
growth and above average achievement. These schools may have at one time been below the 
state average in achievement but their high rates of growth have moved them upward in the 
state distribution. The quadrant could also capture schools that enroll students with solid 
education endowments and also moves them ahead more rapidly than other schools. Working 
clockwise, the lower right quadrant has high growth and low achievement. Positive growth will 
improve a school’s standing over time and — depending on if they grow more quickly than the 
schools in the top quadrant -- possibly move it upwards into the upper right. The lower left 
quadrant holds the schools that are both below average in achievement and produce smaller 
learning gains than their peer schools. If their results remain stable over time, schools with 
higher growth will eclipse them, leaving them even lower in the distribution of schools. Finally, 
schools in the upper left quadrant have above average achievement and growth that lags their 
local peer school. Many of these schools operate in solidly-performing districts and 
communities to provide a distinct alternative to students and families. Families are interested 
in the “something else” and, since the foundation of achievement is already positive, are willing 
to trade off a bit of growth to get it. 


With this guided tour complete, the graphics are populated with actual values in the 
following tables. Two years of school-wide average achievement, drawn from the 2009/10 and 
2010/11 school years, and their associated school-wide average growth were used to locate 
schools within the four quadrants. 
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Table 2: Reading Growth and Achievement 


Low Growth, High Growth, 
High Achievement High Achievement 
Growth -0.15 0 0.15 
(in Standard 
Deviations) 


ee 70th Percentile 


50th Percentile 


30th Percentile 


Low Growth, High Growth, 
Low Achievement Low Achievement 


Quoting from the 2014 report: 


In California, 462 of the 819 charter schools that were included in the chart (about 56 
percent) had positive average growth in reading, regardless of their average achievement 
(this percentage is the sum of the squares in the blue and purple quadrants, the right half 
of the table). About 34 percent of charters had positive growth and average achievement 
above the 50" percentile of the state (i.e., the total for the blue quadrant on the top 
right.) 


About 53 percent of charters perform below the 50" percentile of achievement (the sum 
of the gray and purple in the lower portion of the table). Approximately one in five of 
California charters have positive growth and achievement below the 50" percentile in the 
state, as seen in the lower right, pink quadrant. If those schools continue their trends of 
positive academic growth, their achievement would be expected to rise over time. 


Of concern, however, are the 31 percent of charters in the lower left gray quadrant, which 
represents low growth and low achievement. 
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Table 3: Math Growth and Achievement 


Low Growth, High Growth, 
High Achievement High Achievement 
Growth -0.15 0 0.15 
(in Standard 
Deviations) 


70th Percentile 


50th Percentile 


30th Percentile 


Low Growth, High Growth, 
Low Achievement Low Achievement 


Again quoting from the 2014 report: 


For math, 352 of the 783 charter schools included in the chart (45 percent) had positive 
average growth, as seen in the orange and pink quadrants. Over 27 percent of charters 
had positive growth and average achievement above the 50" percentile (the top right, 
orange quadrant). About 60 percent of charters have achievement results below the 50° 
percentile of the state (the sum of lower half of the table). Of great concern are the 42 
percent of schools which are in the lower left brown quadrant, which represents low 
growth and low achievement. 


A school could be in the lower left quadrant for reading or math or both. Taking the 
union of the schools in either of the two lower left quadrants, a total of 391 schools were 
identified. This sum will be the focus of the remaining analysis. 


Update to the 2014 Report 
As noted above, the data for the quadrant charts was drawn from the 2009/10 and 
2010/11 school years. The performance of the schools that fell into the lower left quadrant 
poses risk for the students they enroll. They also cast a pall over other charter schools who are 


better performing that plays out in public opinion, media attention and political consideration. 


Moreover, these are the schools that the fundamental charter bargain — flexibility for 
accountability — expected to face serious consequences. Either parents would discern the 
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schools’ low performance and move their children to better schools or charter authorizers 
would intervene and force improvements or close the school. 


How well did the charter bargain hold up with these schools? To answer the question 
we followed the 391 schools from 2010 which was the end of the first year of performance 
considered in these charts through the 2016-17 school year. For reasons explained more fully 
at the end of the memorandum, we relied on publicly available school-level data to see if the 
school continued operations despite posting poor performance in the multi-year review. 


Table 4. Number of Low Performing Charter School Closures by School Year 


School Year Number of Closures 
2009/10 2 
2010/11 11 
2011/12 15 
2012/13 16 
2013/14 16 
2014/15 13 
2015/16 6 
2016/17 11 


With the exception of the 2015-16 year, the number of closures appears fairly stable. 
Nearly half of the closures, 44 schools, occurred before the study had been released. The 
steady treatment of these low-performing school closures indicates that the majority of 
closures occur at the end of a charter when the school is reviewed for renewal. 


A total of 90 schools were closed, or 23 percent of the 391 schools that had both low 
academic progress and low achievement. The proportion significantly outpaces the average 
rate of closure of low performing charters, which in a separate CREDO study? was estimated to 
be 5.5% over an equivalent number of years. This comparison suggests that California 
authorizers use school closure more readily than their counterparts in other areas of the 
country. 


* http://credo.stanford.edu/pdfs/Closure FINAL Volume _l.pdf 
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Barriers to Data Access and Evidence Creation 


Updating the report proved difficult for a number of reasons. The State Board of 
Education approved the suspension of annual student testing for federal and state 
accountability reporting for the Spring 2014 assessment. The rationale rested on a temporal 
disconnect between adoption of Common Core Standards in 2010, years ahead of the 2015 
adoption of the Smarter Balanced assessments, designed to those standards. Additionally, in 
2014 a majority of schools lacked the capability to administer the online Common Core-aligned 
pilot assessments. The decision created a disruption in the time series of student testing in 
California that affected the ability to discern academic progress from one year to the next. 


Second, in 2013, the California legislature passed a bill proposed by the Superintendent 
of Public Instruction to prohibit comparison of student or school performance across the two 
testing systems. EdCode 60641(a)(2)) says the California Department of Education and local 
school districts “shall not use a comparison resulting from the scores and results” of the new 
tests “and the assessment scores and results from assessments that measured previously 
adopted content standards.” CDE has extended the restriction to include research and bans 
researchers from employing both test series in any given study. Researcher must choose either 
a longer, more stable series that is not current or a contemporary short series (currently three 
years or two growth periods) that by virtue of being both new and short-lived introduces more 
uncertainty into an analysis. 


Third, in March 2015, the State Board of Education terminated the Academic 
Performance Index — the state’s accountability system. The expectation was that a new 
accountability system, the Local Control Accountability Plan (LCAP), would take its place in short 
order. The first release of school performance occurred in Fall 2017 based on 2016-2017 school 
year performance; several of the indicators were not released pending further refinement or 
longer /more stable data series. However, based on the indicators that were released, the 
State Board of Education revised the cut scores after finding that many more schools received 
Very Low ratings than had been anticipated. 


During this protracted period of development for new budgeting and accountability 
systems, the California Department of Education withheld data — both school-level and student- 
level. In partial defense of CDE’s action, there was a concern that CDE itself did not have full 
working knowledge of the Smarter Balanced data and needed time to test and verify the first 
round of assessments. They also used the time to redesign the data request process and to 
build a web-based portal to automate much of the required researcher activity. Data requests 
were considered for historical data using the former testing and accountability systems, but any 
request for new data was either declined or put on hold. According to correspondence with 
CDE leadership, only 25 requests for data that involve new personal-level data have been 
granted to date. As of the middle of June 2018, CREDO’s request for student data from 2015 
onwards has yet to be granted, after 2 years of negotiation. 
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These barriers to evidence are troubling for three reasons. Most important, state and 
local policy makers are limited in the amount of information at their disposal when faced with 
decisions. Information gaps push decision makers to rely exclusively on softer factors. Second, 
by preventing outsiders the chance to examine programs and policies, independent analysis is 
eliminated from the policy process, resulting in significant harm to policy debate and 
transparency. Finally, the loss of exchange between independent analysts and decision makers 
thwarts a vital source of informed dialogue about policy and program improvement that is so 
critically needed in our public education system. 
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